Decision Tree Discovery

We describe the two most commonly used systems for induction of decision trees for classiication: C4.5 and CART. We highlight the methods and diier-ent decisions made in each system with respect to splitting criteria, pruning, noise handling, and other diierentiating features. We describe how rules can be derived from decision trees and point to some diierence in the induction of regression trees. We conclude with some pointers to advanced techniques, including ensemble methods, oblique splits, grafting, and coping with large data. C4.5 belongs to a succession of decision tree learners that trace their origins back to the work of Hunt and others in the late 1950s and early 1960s (Hunt 1962). Its immediate predecessors were ID3 (Quinlan 1979), a simple system consisting initially of about 600 lines of Pascal, and C4 (Quinlan 1987). C4.5 has grown to about 9,000 lines of C that is available on diskette with Quinlan (1993). Although C4.5 has been superseded by C5.0, a commercial system from RuleQuest Research, this discussion will focus on C4.5 since its source code is readily available.